-
Notifications
You must be signed in to change notification settings - Fork 496
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Equality exclude columns #829
Conversation
I created this PR before the associated issue (#828) was triaged and accepted for development, my apologies! I had forgotten that all PRs must be associated with an issue, oops. For potential reviewers (if the PR makes it that far), I do have one important technical question: I replaced {%- if not compare_columns -%}
{%- do dbt_utils._is_ephemeral(model, 'test_equality') -%}
{%- set compare_columns = adapter.get_columns_in_relation(model) | map(attribute='quoted') -%}
{%- endif -%} by {%- if not compare_columns -%}
{%- do dbt_utils._is_ephemeral(model, 'test_equality') -%}
{%- set compare_columns = adapter.get_columns_in_relation(model) | map(attribute='name') | list -%}
{%- endif -%} I don't know why Edit: Looks like integration tests failed in Snowflake, maybe it's related. Some help would be welcome. |
Update: I figured out what the deal was with Snowflake and quoted columns. I fixed it with a new version that is a bit uglier but seems to work. However it's now failing for BigQuery apparently. Some help would be great 🙏 |
Update: I think I finally figured out how to handle quoting in a satisfactory way (tricky! But I learned something). Note: I tested locally that everything works as expected in Snowflake when some columns are quoted (yuk). I think the PR is good to go! Thanks for reviewing 🙏 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At a glance this looks good! One note is that only one of compare_columns
and exclude_columns
should be accepted (see union_relations for an example)
I'll try and find someone for a more robust review and getting this merged 🎉
Thank you @joellabes ! About forbidding to have both We could easily forbid it explicitly, but imho there's no need. The final set of columns that will be compared is the subset of |
I think this is a duplicate of #765 (although that uses |
@rlh1994 Sorry for the slow response! Indeed it is duplicate, my bad for missing it. Do you know what's the hold up with your PR? Looks like it's been open for more than 6 months. If I compare what I wrote to your PR, I only see 2 tiny differences:
I tried to be very careful about quoting (depending on whether I'm looking forward to your PR being merged and I can close this one. Unless you want to split yours to only add |
@seub I think I don't know of any reasons the PR isn't progressing tbh, I think it's ready to go but just awaiting a review |
resolves #828
This is a:
All pull requests from community contributors should target the
main
branch (default).Description & motivation
This PR adds an optional
exclude_columns
to theequality
test (similar tocompare_columns
).Checklist
star()
source)limit_zero()
macro in place of the literal string:limit 0
dbt.type_*
macros instead of explicit datatypes (e.g.dbt.type_timestamp()
instead ofTIMESTAMP